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[57] ABSTRACT 

A system and method for management of an interface for 
communications systems and devices using a recognizer 
having a limited time duration recognition window, based on 
mapping available time to a spatial representation. The 
system has particular application to management of a speech 
interface for a speech recognizer with a limited time dura- 
tion recognition window. When the speech recognizer is on 
and a recognition window is opened, a timing signal is sent 
to the device and processed to provide spatial representation 
of the time remaining, e.g. in the form of an animation on a 
graphical user interface, or a recognizable pattern of stimu- 
lation using a haptic interface. As the recognition window 
advances and closes, the animation or pattern also changes 
and closes, e.g. diminishes in size, to provide the user with 
spatial information indicative of the time left in the recog- 
nition window. A reset feature allows users to reopen the 
recognition window to delete and correct input, while the 
window is still open, or after closing the window, and before 
the system advances to an erroneous state. The system is 
particularly applicable to speech recognizers having a lim- 
ited recognition window that access speech recognition 
engines resident on a network or at a terminal device. 

29 Claims, 4 Drawing Sheets 
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COMMUNICATION SYSTEM USER 
INTERE\CE WITH ANIMATED 
REPRESENTATION OF TIME REMAINING 
FOR INPUT TO RECOGNIZER 

RELATED APPLICAnONS 

This application is related to U.S. patent application Ser. 
No. 09/062,969 entitled "Server for handling multimodal 
information" to H. Pasternak; and U.S. patent application 
Ser. No. 09/062,970 entitled 'Management of speech and 
audio prompts in multimodal interfaces' to M. French St. 
George, filed concurrently herewith. 

HELD OF THE INVENTION 

This invention relates to management of interfaces for 
telecommunications systems and devices having a limited 
time window for capturing input, with particular application 
to speech recognizers having a limited time window for 
responding to a speech prompt. 

BACKGROUND OF THE INVENTION 

Telecommunications systems with a speech recognition 
capabihty have been in use for some time for performing 
basis tasks such as directory dialling. There are also network 
based speech recognition servers that deliver speech enabled 
directory dialling to any telephone. Typically, speech 
prompted interfaces have been used in telecommunications 
systems in contexts where there is no visual display or the 
user is unable to use visual displays, for example, a con- 
ventional telephone terminal. 

Typically, the speech interface prompts the user when to 
speak by providing a speech prompt, i.e. a recognisable 
phrase or question prompting for user input, or by emitting 
a * speak now* beep after which the speech recognizer is 
turned on, for a limited time window, typically a few 
seconds, during which the user may respond. 

Users of telecommunications equipment employing 
speech recognition systems often report feeling rushed when 
prompted to respond immediately after a beep or other 
audible prompt. 

Part of this rushed feeling may be attributed to a sense that 
the device will stop recognition before the user has com- 
pleted their verbal request because the user receives no 
indication of the time window available to respond after the 
recognizer it turned on or when the recognition window is 
open. The user may find it difficult to know when the speech 
recognizer is on, may talk when the recognizer is off, or may 
become confused by no response. 

Other difficulties may occur if the user does not remember 
what is acceptable input vocabulary or grammar to use. In 
addition to the sense of having to respond right now, current 
speech interface structures do not provide the user with an 
opportunity to rephrase a request, or change their mind 
before waiting for the system to respond. The user's utter- 
ance is accepted and interpreted, and the system advances to 
the next logical state, which may result in an error, for 
example if the user mis-speaks, coughs, or simply makes a 
mistake. Similarly, undue hesitation after a partial response 
may cause the system to move prematurely to the next 
logical state. If this state is not where the user wants to be, 
the user must navigate back to the previous state and restate 
the request. 

Currently, the best recognizers in use have a 90 to 95 
percent recognition performance under optimum conditions, 
and a noisy background environment, other speakers, user 
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accents, the user speaking to softly, may adversely affect 
recognition performance. 

When conditions are not optimum, additional dialogue 
may assist. For example, the recognizer may give repeat 
^ instructions, or provide additional instructions. 
Nevertheless, using speech to provide additional informa- 
tion is slow. Consequently the user may perceive an exces- 
sively long wait for the system to reset and issue a new 
prompt. Typically, speech is perceived as fast for input, and 
10 slow for output. 

Many users report becoming fiaistrated with using inter- 
active voice response (IVR) systems offering many choices 
or a multi level menu system of choices. The user may forget 
long lists of instructions, or become confused or lost in a 
complex speech appfication. 

User difficulties in interacting with these systems repre- 
sent some reasons such speech interfaces have not yet 
gained as widespread acceptance as they might. 

Older systems which also provide a graphical user 
interface, i.e. a screen display, with a speech interface, have 
been discrete non-integrated techniques That is the system 
may use either a touch input or a speech input, but not both 
simultaneously. 

25 To overcome the inconvenience of switching between 
discrete appUcations offering different modes of interaction, 
systems are being developed to handle more than one type 
of interface, i.e. more than one mode of input and output, 
simultaneously. In the following description the term input/ 

3Q output modality refers to a sensory modality relating to a 
user's behavioiu" in interacting with the system, i.e. by using 
auditory, tactile and visual senses. Input/output modes refer 
to specific examples of use of these modalities. For example 
speech and audio input/output represent an auditory modal - 

35 ity; use of a keypad, pen, and touch sensitive buttons 
represent a tactile input modafity, and viewing a graphical 
display relies on the visual modahty. 

An example of a multimodal interface is 08/992,630 
entitled "Multimodal User Interface", filed Dec. 19, 1997, to 

40 Smith and Beaton, which is incorporated herein by refer- 
ence. This application discloses a multi-modal user interface 
and provides a telecommunications system and methods to 
faciUtate multiple modes of interfacing with users for 
example, using voice, hard keys, touch sensitive soft key 

45 input, and pen input. This system provides, e.g. for voice or 
key input of data, and for graphical and speech data output. 
The user may choose to use the most convenient mode of 
interaction with the system and the system responds to input 
from all modes. 

50 Thus, interfaces for commimications devices and com- 
puter systems are becoming increasingly able to accept input 
and provide output by various modes. 

For example, current speech recognition interfaces may 
be used in association with an visual display showing an 

55 icon that indicates current word recognition state. These 
icons change visually when the recognition state changes 
from listening to not-listening. For example, a "talk now** 
icon may be displayed in the corner of the screen. While 
these icons indicate to the user that the speech recognizer is 

60 on, the icons do not overcome the users perception of 
urgency to talk before the window closes. Also, as men- 
tioned above if an error is made, or speech input is inter- 
rupted by extraneous background noise, the system waits 
until the 'talk now' or recognition window closes, and 

65 advances to the next logical state to recover fi^om such an 
error, before issuing a new prompt and reopening the rec- 
ognition window. 
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There also exist natural language speech interfaces that According to yet another aspect of the invention there is 

are always on, which preclude the need for beeps that inform provided a communications system comprising: 

the user of when to start talking. The user may speak at any ^ speech interface for accessing a speech recognizer 

time, and the recognizer wiU always be ready to listen. operable for receiving speech input during limited time 

Currently this type of recognition is not yet widely distrib- 5 ^cognition window 

uted and used. These more advanced speech recognizers ^ ] , , . ^ , 

currently rely on a network based speech recognizer to °^eans for mapping available iunt of the recogmtion 

provide the necessary processing power. Thus in the fore- window to a spatial reprcsentaUon m animated form using 

seeablefuhire, this type of advanced speech recognition will one of a graphical modality, haptic modality or auditory 

co-exist with simpler forms of recognition that require a modality. 

limited duration 'time to talk' window, or recognition win- Thus where the interface is a speech interface and the 

dow. animation is an animation displayed on the graphical 

SUMMARY OF THE INVENTION interface, the animation itself may provide the touch sensi- 

™ . , , .J , , live region which shrinks as the recognition window closes. 

Thus, the present mvention seeks to provide a system and j u • u • ui u *u * . *u <*• * *n 

J^c *r ' * c c ' and which IS operable by the user to reset the tune to talk 

method for management of an interface for a commumca- 15 .. ^ ^ . .-i -r 

tions systems or devices including a limited time duration recognition window An appropna e graphical ammation for 

window for accepting input, and particularly for manage- ^ ^P^^^^ interface is a speech baUoon. 

ment of a speech interface for a speech recognizer with a Preferably, the system comprises means for resetting the 

limited duration * time-to-talk' window, which avoids or recognition window, which comprises a tactile input means 

reduces some of the above mentioned problems. 20 associated with the animation displayed on the graphical 

Therefore, according to a first aspect of the present iiser interface, e.g. a button, 

invention there is provided a communications system com- For example when the animation is provided by a touch 

prising: sensitive region of the graphical user interface, the means for 

an interface operable for receiving input during limited resetting the recognition window comprises an associated 

time recognition window, 25 touch sensitive region of the graphical user interface. 

means for mapping available time of the recognition Where speech input has ahready been captured, the reset 

window to a spatial representation in animated form using means also functions to delete speech input when the speech 

one of a graphical modality, haptic modality or auditory recognition window is reset. 

modality. According to another aspect of the present invention there 

For example the spatial representation comprises one of a 30 ^ provided a method of providing user feedback and control 

two dimensional representation and three dimensional rep- for a communications device for accessing a recognizer for 

resentation which diminishes in size as the available time receiving input while the recognizer is operable during a 

diminishes, thereby providing a user with an indication of limited time duration recognition, comprising the steps of: 

the available time to provide input. g^^, prompting the user for input; 

Where the means for mapping comprises a graphical user ^^^^ recognizer for a limited time duration 

mterface, preferably the spatial representation comprises recognition window; 

graphical information in the form of an animation, which j . -i ' ■ • 1 i_i . 

J. . . . . . . 1 LI J- • • u . and while the recognizer IS on, mappmg available time to 

dimimshes m size as the available time diminishes. The rate ^. , ^ .'^^ . ' , /^^ - r i.- 

r . . • *• i_ I- 1- a spatial representation in ammated form m one of a graphi- 

01 size change of the animation may be linear or non-linear, , j i-: u j 1-. j j.-. j i-. 

. J f . « . - . 40 cal modality, haptic modality and auditory modality, 

as required, to influence the users perception of the remam- ^ ^ » r j j j 

ing time available Where the interface comprises a graphical user interface 

Alternatively, the means for mapping comprises a haptic method comprise providing a spatUl representation in 

interface and the spatial representation comprises an ani- "'^ ".^ ^ gr^ph.ca animation mdicative of the time 

mated pattern of haptic stimulation, e.g. produced by a '«""»n'ng recognition window and preferably com- 

wristband stimulator 45 pnses displaymg an ammauon which dimmishes in size as 

f . . ,. the time remaining in the recognitioi) window decreases. 

Alternatively, the means for mappmg comprises an audi- ^ . ,,. . , , ^ , ,. .. 

tory interface the spatial representation comprises a time '"'^'hod is particularly useful for communications 

dependent auditory pattern. ''^^S ^ ^1*^'='^ mterface for accessing a speech 

TTiese alternative modalities may be combined, for '«'=°g°i^"; the speech recognizer is operable for 

example to provide both graphical and haptic feedback to '° ^^^^ '"P"* ^"""^^ * ^'"^"^ '"^^ ^'=°e.nition 
the user regarding available lime in the recognition window. 

Preferably, the system also provide a means for resetting BeneficiaUy while the speech recognizer is on, the method 

the recognition window by the user, and reinitiating the comprises generatmg a tmiing signal representative of the 

animation. For example, the reset means may be a key or ^"'^ remammg m the recogniUon wmdow, 

button associated with the animation. Alternatively, a touch controUmg the animation usmg the limmg signal and 

sensitive region may be associated with the animation, e.g a at each time increment of the liming signal capturing 

touch sensitive button associated with the animation. speech input, and checking for user input to a reset means. 

According to another aspect of the invention there is and when user input to the reset means is captured, 

provided a communications device comprising: go resetting the recognition window, deleting captured speech 

an speech interface for accessing a speech recognizer input, and reinitiating the timing signal and the animation, 

operable for receiving speech input during limited time and when the timing signal corresponds to the end of the 

recognition window, and recognition window, capturing aggregated speech input and 

means for mapping available time of the recognition closing the recognition window, 

window to a spatial representation in animated form using 65 Another aspect of the present invention provides software 

one of a graphical modality, haptic modality or auditory on a computer readable medium for carrying out these 

modality. methods. 
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Conveniently, in use of a speech interface, the graphical FIG. 8 shows a flow chart setting out the steps of a method 

information is an animation displayed on the graphical user of providing interface management according to an embodi- 

interface which changes as the recognition window closes ment of the present invention, i.e. user feedback and control 

with time. For example the animation may be a speech of a speech interface using a multimodal interface of the 
balloon, or other appropriate icon, which diminishes in size 5 mobile telephone shown in FIGS. 2 and 3. 
as the recognition window closes. The animation provides 

the user with a graphical representation of the time remain- DETAILED DESCRIPTION OF THE 

ing to talk. Movement on the screen also has the advantage INVENTION 

of drawing the users attention to the closing recognition A schematic block diagram of a communications network 

^° 10 is shown in HG. 1 and represents a GSM switching 

Thus the user receives a visual indication of the time services fabric 20 and a network services provider 40 

remaining in the recognition window of the speech associated with a plurality of communications terminals, for 

recognizer, and is provided with the option to reset the example a mobile telephone 100, and other wired or wireless 

recognition window, for example, to correct a speech input communications devices and terminals represented sche- 
error. Thus the user receives feedback regarding the opera- 15 matically by units 110, 120, 130, 140 and 150. 

tion of the speech recognizer, and is provided with an ^^^^Ue phone 100 according to a first 

opportunity to control opemng of the recognition window, embodiment of the present invention is shown enlarged in 

e.g. to correct errors. Pjq 3, and is provided with a multitasking graphical user 

Beneficially, by using a multitasking, multimodal user interface, similar to that described in copending U.S. appli- 

interface, and combining a touch sensitive display to the ^0 ^^^.^^ entitled "Multimodal User Interface" filed 

interface, an animated graphic can also serve as a reset d^c. 19, 1997, 08/992,630 to Smith and Beaton, which is 

button that the user may touch, to re-open a closing window. incorporated herein by reference. This is a multi-modal user 

This user behaviour resets the recognition window and interface which provides a telecommunications system and 

allows the user to correct or alter a spoken command to a methods to facihtate muUiple modes of interfacing with 

speech interface without having to wait for the system to users for example, using voice, hard keys, touch sensitive 

achieve its next state before returning to correct an input. soft key input, and pen input. This system provides, e.g. for 

Reset means allows the user reset the time window at will. voice or key input of data, and for graphical and speech data 

If not reset the animation stops when the time reaches the output. Thus the user may choose to use the most convenient 

time duration Tw and the recognition window closes, after mode of interaction with the system and the system responds 

capturing aggregated speech input collected while the win- to input from all modes. 

dow is open. ^ shown in FIG. 2, mobile telephone unit 100 comprises 

Provision of an animated graphical output, with or with- a body 200 which carries a display screen 210 for the 

out haptic and auditory feedback, indicative of time graphical user interface, which may include touch sensitive 

available, provides the user with additional timing informa- buttons 212; a conventional keypad 220 and other hard keys 

tion. Consequently user feedback and control of the inter- 222; speaker 240 associated with the speech interface to 

face or speech interface is improved. providing speech prompts as shown schematically by 230 to 

BRIEF DESCRIPTION OF THE DRAWINGS illustrate the various modes of interaction which may be 

^ . . .„ , , , . selected by a user. Thus the system provides for conven- 

nie invention wiU now be described in greater detail with .j^^^, ^ ^ ^j^^, .^^^^^^^ ^^^^ 

reference to the attached drawings wherein: « ^^^^ ^^^^^ ^^j, ^^y^^ ^ ^p^^^j, j^,^^^^^ ^„,p^,. 

HG. 1 shows a schematic block diagram of a communi- ^p^^ j^jjo prompts, and a speech recognizer to 

cations network comprising a mobUe telephone having a jccept and interpret speech input. OptionaUy, the graphical 

multitasking graphical user interface consistent with an ^^ei interface may support pen input to aUow written input 

embodiment of the present invention; ^ sensitive area of the screen. 

HG. 2 shows schematically a diagram of mobile tele- ^he device 100 accoiding to the first embodiment of the 

phone of a first embodiment for operation m the network of p^^^^, invention comprises means for mapping available 

' ' time of the recognition window to a spatial representation in 

FIG. 3 shows part of the mobile telephone of FIG. 2 one of a graphical, haptic or auditory modality. This embodi- 

showing on an enlarged scale detail of the touch sensitive ^^^^ example using a graphical modality for providing 

graphical display with pen input capability, during operation ^^cr feedback and control of a 'time to talk' recognition 

of a YeUow Pages directory apphcation using a method window of the speech interface. In FIG. 2, and in the 

accordmg to an embodiment of the invention. enlarged schematic of the screen display shown in FIG, 3, 

HGS. 4A, 4B, 4C, and 4D show schematicaDy part of the near the bottom of the display screen 210, the user feedback 
display shown in FIG. 3, showing on an enlarged scale, to 55 means is an indicator which takes the form of an appropriate 

show the time to talk indicator and controller displayed by animated icon, or animation 216, as will be described below, 

the graphical interface as the time window of the speech and is associated with a touch sensitive button 218. The 

recognizer advances; button 218 functions to control and reset the speech recog- 

FIG. 5 shows schematically part of a graphical user nizer. The lower part of the screen 210 is shown enlarged in 

interface representing an second embodiment of the indica- FIGS. 4A, 4B, 4C and 4D to show how the animation 

tor and controller, changes, i.e. shrinks, as the lime to talk window of the 

FIG. 6 shows schematically part of a graphical user speech recognizer is opened and closes, 

interface representing an third embodiment of the indicator Since speech and word recognition systems have limited 

and controller; recognition windows, when the speech recognizer is turned 

FIG. 7 shows schematically part of a graphical user 65 on and a recognition window is opened, a timing signal is 

interface representing an fourth embodiment of the indicator sent to the device that starts the indicator, for example 

and controller; displaying the animation shown on a graphical display 
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represented in FIGS. 4A to 4D. The signal contains infer- A method in accordance with the first embodiment of the 

mation regarding the timing of the animation, and as the present invention, with reference to the portable wireless 

recognition window advances and closes, the animation also terminal 100 shown in FIGS. 2 to 4, will be described in 

changes and closes. In this example the animation is a more detail with reference to the flow chart shown in FIG. 
speech balloon which diminishes in size as the time to talk 5 8. 

window closes. The timing signal contains information Inoperationof the device, the use may pick up the device, 

regarding the liming of the animation. As the recognition and cause activation of the device either automatically upon 

window closes , the animation also closes. picking up the device, or for example by pressing a button. 

The rate at which the animation diminishes in size is speech recognizer is thereby switched on, and 
controlled by the timing signal. The rate of change of the lo p^mpts the user for input (A). The prompt may be an audio, 

animation may be linear or non-linear with respect to the ^^^^^ speech or an 'earcon', that is a recognizable sound 

time available. While the user does not receive a count g^^^h as a characteristic 'beep'. The speech recognizer (SR) 

down. I.e. it is not a numenc display of the tune remaming, window is himed on (B) for a specific recognition window 

the animation provides a spatial representation of the time j^j^^ ^nd a timing signal is initiated at T-0, which is 
remaining, which allows the user to gauge how much time 15 indicative of the time remaining, Tw, in the recognition 

is left. The user receives suflBcient visual feedback on timing window, 

information to provide and indication of the progress of the a** i-^ *j u-* ic/^- 

. J , ^ . . ,1. - J -11 At time T=0, an aggregated speech mput sample St^O is 

window, and warning when the recognition wmdow will ^ ^ /r^\ j * * n • • * j 

1 ^ * c u f*u ' ^ u J* captured (C) and a Ume-to-talk ammation assoaated with 

close. The rate of change of the anmiation may be used to *• ■ - i • j- i j *u j *u 
. „ * f 1 i_i J -,n the timmg signal is displayed on the screen and the anima- 

innuence the users perception of the available time and the 20 . .v. j /t^v t-u i_ u h • .u 

* 4 u' u .u -J • 1 • 110° IS started (D). The speech balloon is representmg the 

rate at which the window IS closing. . .V . . . ,i , - j • 

, . , . . ^ ^ , ^ ^ , speech recognition wmdow, or time to talk wmdow, is 

In the example shown m FIGS 2. 3 and 4A to 40, the ^^^^ maximum size as shown in HG. 4A. 

ammation IS a graphical speech balloon. Touching the button , . , . . ... 

218 associated with the animation, i.e. on an touch sensitive ^^^^^ ^P^*^^,^ ^"^^ ^^^^^.^ if^'^^o^' 
adjacent region of the screen display, sends a signal to the 25 aggregated speech input sample is coUected S^S^+S, After 

recognizer to delete any verbal signal associated with a each speech mput is sampled a check is made to determine 

previously opened recognition window in this state, and ^P^^'^h recognizer wmdow is open, i.e. if T^Tw. 

resets the timing signal, and the animation, indicating that When the time value is checked, if the speech recognizer 

the user may restate their request. recognition window has closed at T=Tw the aggregated 

Thus, the user is provided with information regarding the speech sample is sent for speech recognition (G). 

time left in the recognition window, and an undo or reset If the time window is still open T<Tw, the recognizer 

feature is also provided. The reset feature allows users to checks whether the user has issued a reset command, i.e. 

correct speech utterances while the window is still open, or pressed the reset button, to reset the recognition window 

after closing the window, and before the system advances to (H). If not, the timing signal advances, the collection of the 

an erroneous state. next input speech sample is commenced (E). The animation 

TTius the speech interface is provided with animated continues to advance, diminishing in size continuously as 

graphical support, to indicate the time duration of an avail- the timing signal advances (see FIGS. 4B to 4D), lo provide 

able recognition window. the user with a further visual indication of the diminishing 

OptionaUy, haptic support is provided in addition lo available time to talk left in the recognition window, 

graphical output, for example by providing a spatial repre- When T-TXv, the aggregated speech sample is captured, 

sentation of the time remaining through a haptic pattern the recognition is closed, i.e. the recognizer is turned off, and 

generated by an array of stimulators carried on an associated the animation ceases. 

wristband of the communications device. If at any time during the recognition window, the user 

While in this first embodiment, the balloon animation and resets the speech recognizer, any aggregated speech input 

balloon button are separate entities, advantageously, accord- that has been captured is erased, and a prompt for new 

ing to a second embodiment, the animation is provided on a speech input is issued, e.g. using a simple *earcon*. The 

touch sensitive area of the display, i.e. the animation itself is timing signal is reset, (I) and the animation is reset to the 

touch sensitive and functions as a soft button providing the time zero display as shown in FIG. 4A. 

reset function, as shown schematically in FIG. 5. The method of operation of the indicator and controller is 

Alternatively a soft key implementation of the indicator, generally applicable to all devices that access speech rec- 

i.e. the latter balloon button could alternatively is by a hard ognilion engines, whether these are implemented locally, or 

key button version, as shown schematically in FIG. 6. The on a network server. 

animation illustrated above shrinks in length as the available The system and method described above was developed 

time diminishes, but the animation may alternatively take particularly to facilitate interaction with a speech recognizer, 

any convenient form, and inanother embodiment shown in The system and method in general is not limited to speech 

FIG. 7, the animation shrinks in 2 dimensions. These are interfaces, but may also be applied lo other interfaces 

simply given as examples, and any convenient form of operating with a limited lime window for receiving input, 

animation may be used. Another example may be a system for receiving and recog- 

The user is provided with graphical and dynamic infor- 50 nizing security information e.g. user id and password 

mation regarding the tune left in the recognition window. information, to provide access lo an information system. 

The reset, or undo feature allows users to correct speech wherein a lime limit for access is provided, for example to 

utterances before the system advances to an erroneous state. deter unauthorized entry. 

Speech recognition may reside on the network, or at the In another example (not shown) the haptic animation 
terminal device, and the indicator and controller is appli- 65 mentioned above may be provided instead of graphical 

cable to speech and word recognition systems having limited information regarding the available lime in the recognition 

recognition windows. window. In another example (not shown), the animation is 
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auditory, i.e. a pattern of sound representative of the avail- means for mapping available time of the recognition 

able time may be generated by stereo speakers of the window to a spatial representation in animated form 

communications device. using one of a graphical modality, haptic modality or 

While the systems and methods described above may be auditory modality, 

implemented by any convenient software and hardware 5 12, A system according to claim 11 wherein the means for 

configuration for local device or network server implemen- mapping comprises a graphical user interface and the spatial 

tation of a speech interface, and the method is intended to be representation comprises a graphical animation, 

platform independent. For Internet compatible communica- x3. A communications system according to claim 12 

tions devices, the cunent application is a preferred imple- comprising means for resetting the recognition window, 

mentation of the current appUcation uses a Java web server, ^4 ^ communications system according to claim 13 

such as one designed for this application as describ^ in the ^^^^^^^ ^^^^ recognition window 

above referenced copending U.S. patent application Ser. No. comnrises a tactile innut means Associated with the anima- 

09/062,969, entitled "Server for handling multimodal infor- ^omprises a tactile mput means associated with the anuna 

mation" to H. Pasternak filed concurrently herewith, and lio" /i^layed on the graphical user interface, 

incorporated herein by reference. This server was developed J^" communications system according to claim 13 

for handling information in different modal forms associated wherein the anmiation is provided by a touch sensiUve 

with respective input/output modalities of a multimodal ^^^^^^ S^^P^!^.^' user mterface, and the means for 

interface, and may be used to implement the systems and ^^^^"^8 recognition wmdow cooapnses an assoaated 

methods described herein. ^°"^3* ^8^°° graphical user interface. 

. , .„ ... . f • u lo. A system according to claim 11 wherein the spatial 

Although specific embodiments of the invention have „ ^ \. - i 

u J -u J • J * 1 11 u * . 1 11 J ' 20 representation comprises one of a two dimensional repre- 

been described m detail. It will be apparent to one skilled in a*u a- 1 * u- u^- • 

.u -* *u . - \> ji a J * *u u J- . sentation and three dimensional representation which dimm- 

the art that variations and modifications to the embodiments . . 1 1.1 j- • • u 

. , ..u* *u c *u c ^^ ' i • ishes in size as the available time dimmishes. 

may be made withm the scope of the followmg claims. 1-. ^ _j-.i-nu -.u c 

/.„ . ■ 1 • J • 17. A system according to claun 11 wherein the means for 

What IS claimed is: • • u *• • . _f j *u 1 

^ . . , . . mappmg comprises a haptic mterface and the spatial repre- 

1. A commumcations system compnsing: . • • . , ru .1 

. . / . . . ^ ,25 sentation compnses an animated pattern of haptic stunula- 

an interface operable for receiving input during limited j-^^j 

time recognition window, Asy^itm according to claim 11 wherein the animation 

means for mapping available Ume of the recognition diminishes in size as the available time diminishes, 

window to a spatial representation in animated form 19 ^ system according to claim 18 wherein the rate of 

using one of a graphical modality, haptic modality or 30 change in size of the animation is Unear. 

auditory modality. 20. A system according to claim 18 wherein the rate of 

2. A system according to claim 1 wherein the spatial change in size of animation is non-linear, 
representation comprises one of a two dunensional repre- 21. Asystem according to claim 11 wherein the animation 
sentation and three dimensional representation which dimin- indicative of the operation of the recognition window, and 
ishes in size as the available time diminishes. 35 animation changes during the operation of the recogni- 

3. A system according to claim 1 wherein the means for ^^^^ window to indicate the remaining time duration of the 
mapping compnses a graphical user interface and the spatial recognition window 

representation comprises graphical information in the foraa 22. A method of providing user feedback and control for 

of an animation. ^ communications device for accessing a recognizer for 

4. Asystem according to claim 3 wherein the animation ^ receiving input while the recognizer is operable during a 
dmainishes in size as the available time diminishes. ji^jted time duration recognition, comprising the steps of: 

5. A system according to claim 4 wherein the rate of r • 
change of the animation is Unear. promptmg the user for mput; 

6. A system according to claim 4 wherein the rate of turning on the recognizer for a limited tune duration 
change of animation is non-linear. 45 recognition window; 

7. A system according to claim 1 wherein the means for and while the speech recognizer is on, mapping available 
mapping comprises a haptic interface and the spatial repre- tiine to a spatial representation in animated form in one 
sentation comprises an animated pattern of haptic stimula- of a graphical modality, haptic modality and auditory 
tion. modality. 

8. A system according to claim 1 wherein the means for jq 23. A method according to claim 22 wherein the interface 
mapping comprises an auditory interface the spatial repre- comprises a graphical user interface and the method com- 
sentation comprises an time dependent auditory pattern. prises providing a spatial representation in the form of an 

9. Asystem according to claim 1 comprising reset means graphical animation indicative of the time remaining in the 
for resetting the recognition window and reinitiating the recognition window. 

animation. 24. A method according to claim 23 comprising display- 

10. A communications device comprising: i^g an animation which diminishes in size as the time 
an speech interface for accessing a speech recognizer remaining in the recognition window decreases. 

operable for receiving speech input during limited time 25. A method of providing user feedback and control for 

recognition window, and ^ communications device having a speech interface for 

means for mapping available time of the recognition 60 ^"^^ssing a speech recognizer, the speech recognizer being 

window to a spatial representation in animated form operable for receiving mput during a lunited time duration 

using one of a graphical modality, haptic modality or recognition wmdow, compnsing the steps of: 

auditory modality. after prompting the user for input; 

U. A communications system comprising: turning on the recognizer for a limited time duration 

a speech interface for accessing a speech recognizer 65 recognition window; 

operable for receiving speech input during limited time and while the speech recognizer is on, mapping available 

recognition window, time to a spatial representation in animated form using 
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one of a graphical modality, haptic modality and audi- 
tory modality. 

26. A method according to claim 25 wherein the interface 
comprises a graphical user interface and the method com- 
prises providing a spatial representation in the form of an 
graphical animation indicative of the lime remaining in the 
recognition window. 

27. A method according to claim 26 comprising display- 
ing an animation which diminishes in size as the time 
remaining in the recognition window decreases. 

28. A method according to claim 27 comprising, 
while the speech recognizer is on, generating a timing 

signal representative of the time remaining in the 

recognition window, 
controlling the animation using the timing signal and 
at each time increment of the timing signal capturing 

speech input, and checking for user input to a reset 

means, 

and when user input to the reset means is captured, 
resetting the recognition window, deleting captured 
speech input, and reinitiating the timing signal and the 
animation. 
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and when the timing signal corresponds to the end of the 
recognition window, capturing aggregated speech input 
and closing the recognition window. 
29. Software on a computer readable medium for carrying 
out a method of providing user feedback and control for a 
communications device having a speech interface for 
accessing a speech recognizer, the speech recognizer being 
operable for receiving input during a limited time duration 
10 recognition window, comprising the steps of: 

after prompting the user for input; 

txirning on the recognizer for a limited time duration 
recognition window; 

and while the speech recognizer is on, mapping available 
time to a spatial representation in animated form using 
one of a graphical modality, haptic modality and audi- 
tory modality. 
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